Run a parallel pre-check before updating toolchains#4752
Run a parallel pre-check before updating toolchains#4752FranciscoTGouveia wants to merge 3 commits intorust-lang:mainfrom
Conversation
src/cli/common.rs
Outdated
| future::join_all(channels.iter().map(|(_, d)| d.show_dist_version())) | ||
| .await | ||
| .into_iter() | ||
| .map(|v| v.map_or(true, |opt| opt.is_some())) | ||
| .collect() | ||
| }; |
There was a problem hiding this comment.
Is there a cleaner way to achieve this?
There was a problem hiding this comment.
Am I reading it right that now for every toolchain that needs to be updated, the update is checked twice?
There was a problem hiding this comment.
Yes, I forgot to add a comment about this, sorry.
I see two possible ways to proceed:
- Accept the cost of performing a double check. In practice, since the first check is done concurrently in a batch, the overhead should be relatively small.
- Do a small refactor, starting in
DistOptions::install_into(), to allow skipping the second check.
There was a problem hiding this comment.
@FranciscoTGouveia I think we should try to avoid double checking if the changes involved are not super disruptive 🙏
There was a problem hiding this comment.
I have been thinking about this, and my initial idea was to add a new manifest field (of type ManifestV2) to the DistOptions struct; however, I believe that would change the semantics quite a bit, since it would store more than just "options".
A direction I am more inclined toward is to reuse the manifest returned by show_dist_version(), which is currently discarded since we only call .is_some() on it.
Pairing this with a potential new install_with_manifest() method in InstallMethod (alongside the existing install()) would allow us to skip re-downloading the manifest.
What do you think?
There was a problem hiding this comment.
A direction I am more inclined toward is to reuse the manifest returned by
show_dist_version(), which is currently discarded since we only call.is_some()on it.
Pairing this with a potential newinstall_with_manifest()method inInstallMethod(alongside the existinginstall()) would allow us to skip re-downloading the manifest.
What do you think?
Sounds pretty good to me.
There was a problem hiding this comment.
A direction I am more inclined toward is to reuse the manifest returned by
show_dist_version(), which is currently discarded since we only call.is_some()on it. Pairing this with a potential newinstall_with_manifest()method inInstallMethod(alongside the existinginstall()) would allow us to skip re-downloading the manifest. What do you think?
@FranciscoTGouveia Yeah I agree with that. It'd be nice if the old installation method can be refactored to use the newly-introduced one by fetching the manifests before the call. I imagine there shouldn't be a lot of dependency issues in doing so?
18df18c to
d6cc274
Compare
src/cli/common.rs
Outdated
| join_all(channels.iter().map(|(_, d)| d.show_dist_version())) | ||
| .await | ||
| .into_iter() | ||
| .map(|v| v.map_or(true, |opt| opt.is_some())) |
There was a problem hiding this comment.
I'm confused by this logic. show_dist_version() seems to merely yield the latest available version, but that a DistributableToolchain has a manifest with a rust component with a version is_some() doesn't mean that it's newer than the installed version?
There was a problem hiding this comment.
Sorry, I do not think I fully understood your comment.
The rationale for using show_dist_version() comes from how it's used in rustup check.
As I understand it, show_dist_version() only returns Some when the hashes do not match (i.e., there is an update available), which aligns with what we need here.
In this setup, for toolchains that do not require updating, we would never reach .get_rust_version(), since .dl_v2_manifest() returned None.
Apologies if I've misunderstood or overlooked anything.
There was a problem hiding this comment.
As I understand it,
show_dist_version()only returnsSomewhen the hashes do not match (i.e., there is an update available), which aligns with what we need here.
You're completely right. Might be good to add a docstring for show_dist_version() and maybe consider renaming it.
There was a problem hiding this comment.
Sounds good, I will take care of it. :)
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
There was a problem hiding this comment.
Apologies, could you clarify where you were suggesting putting updated_dist_manifest()?
I have renamed it to fetch_dist_manifest, which I believe is more accurate than the previous show_dist_version().
For reference, this is the updated code, where .is_some() has been replaced with .unwrap_or(None).
There was a problem hiding this comment.
@FranciscoTGouveia Sorry for not being super clear. I was only referring to the renames. show_dist_manifest() may be replaced by updated_dist_manifest() since it will only return a value when there is actually an updated dist manifest, right?
For show_dist_version() I think it'd be better to just inline it as per #4752 (comment).
d6cc274 to
5332ad0
Compare
5332ad0 to
58defd5
Compare
58defd5 to
3afbe36
Compare
src/cli/common.rs
Outdated
| join_all(channels.iter().map(|(_, d)| d.show_dist_version())) | ||
| .await | ||
| .into_iter() | ||
| .map(|v| v.map_or(true, |opt| opt.is_some())) |
This comment was marked as duplicate.
This comment was marked as duplicate.
Sorry, something went wrong.
| .await | ||
| } | ||
|
|
||
| pub async fn show_dist_version(&self) -> anyhow::Result<Option<String>> { |
There was a problem hiding this comment.
Since this has a single caller, maybe this should be inlined into its caller instead?
(If not, should do top-down order such that show_dist_version() comes before fetch_dist_manifest().)
| self.install_with_manifest(None).await | ||
| } | ||
|
|
||
| pub(crate) async fn install_with_manifest( |
There was a problem hiding this comment.
Suggest just adding the argument to install() instead of adding a wrapper, since calling it with None is fairly trivial.
| pub(crate) async fn install_into( | ||
| &self, | ||
| prefix: &InstallPrefix, | ||
| manifest: Option<(ManifestV2, String)>, |
There was a problem hiding this comment.
I think it's fairly unclear with all the places you're passing around (ManifestV2, String) (or (Manifest, String)) what the relation is between those or what the String contains. Perhaps it's worth wrapping these in struct type with named fields?
There was a problem hiding this comment.
@djc It is worth noting that this is the return value of .dl_v2_manifest(). OTOH maybe it is worth it to extract that in a separate commit though.
| join_all(channels.iter().map(|(_, d)| d.fetch_dist_manifest())) | ||
| .await | ||
| .into_iter() | ||
| .map(|r| r.unwrap_or(None)) | ||
| .collect(); |
There was a problem hiding this comment.
So this is sort of just silencing/ignoring errors? Is that what we did before?
Instead of the type annotation, I would use turbofishing and leaving any inner types implied, as in collect::<Vec<_>>().
| targets: &[&str], | ||
| fetched: &mut String, | ||
| cfg: &Cfg<'_>, | ||
| prefetched_manifest: Option<(ManifestV2, String)>, |
There was a problem hiding this comment.
Does this actually need to take ownership ownership of the manifest, or could it make do with a reference?
Fixes #4747.
Instead of sequentially checking each toolchain for updates,
rustup updatenow first checks all toolchains concurrently and only then performs the updates (if needed).This brings two advantages:
rustup updateis now ~2x faster (see below);nightlyneeds updating.Benchmarks were run using
hyperfine(20 iterations on a 50 Mbps connection) on a setup with 11 toolchains, and no regressions were observed.For the no-op
rustup update, this patch was 2.2x faster; for cases where a single toolchain was updated, there was no measurable difference.CC @epage